Maria Zazpe Quintana
Alba Rodríguez Berenguel
Tras haber realizado los modelos (DummyClassifier, Random Forest, Decision Tree, LightGBM y XGBoost), y además, haber calculado sus métricas de evaluación (Precision, Recall, F1, Accuracy, matriz de confusión, curva ROC, curva lift y curva de ganancia), vamos a compararlos y decidir cuál de ellos va a ser el modelo elegido para la predicción, y por tanto, el que optimizaremos mediante una cross validation para elegir los parámetros que más nos convienen. Por lo tanto, vamos a seguir la siguiente estructura:
Con lo ya visto en cada uno de los notebooks anteriores, vamos a hacer una tabla a modo de resumen con las medidas de evaluación principales para tener una vista general.
# We create a dictionary with the evaluation of the models
models_comp = {'Modelo': ['Base (Dummy)', 'Random Forest', 'Decision Tree', 'LightGBM', 'XGBoost'],
'Accuracy (True Positives)': ['0.72', '0.65', '0.69', '0.74', '0.74'],
'False positives':['0.28', '0.35', '0.30', '0.26', '0.26'],
'Precision(0)': ['0', '0.41', '0.44', '0.62', '0.62'],
'Precision(1)': ['0.72', '0.80', '0.76', '0.75', '0.75'],
'Recall(0)': ['0', '0.54', '0.30', '0.17', '0.18'],
'Recall(1)': ['1', '0.70', '0.85', '0.96', '0.96'],
'F1(0)': ['0', '0.46', '0.35', '0.27', '0.28'],
'F1(1)': ['0.84', '0.74', '0.80', '0.84', '0.84']
}
# We convert the dictionary to a df
models_comp = pd.DataFrame(models_comp)
models_comp
| Modelo | Accuracy (True Positives) | False positives | Precision(0) | Precision(1) | Recall(0) | Recall(1) | F1(0) | F1(1) | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | Base (Dummy) | 0.72 | 0.28 | 0 | 0.72 | 0 | 1 | 0 | 0.84 |
| 1 | Random Forest | 0.65 | 0.35 | 0.41 | 0.80 | 0.54 | 0.70 | 0.46 | 0.74 |
| 2 | Decision Tree | 0.69 | 0.30 | 0.44 | 0.76 | 0.30 | 0.85 | 0.35 | 0.80 |
| 3 | LightGBM | 0.74 | 0.26 | 0.62 | 0.75 | 0.17 | 0.96 | 0.27 | 0.84 |
| 4 | XGBoost | 0.74 | 0.26 | 0.62 | 0.75 | 0.18 | 0.96 | 0.28 | 0.84 |
# Libraries
import pickle
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
import scikitplot as skplt
from xgboost import XGBClassifier
from sklearn.metrics import ConfusionMatrixDisplay, fbeta_score, plot_confusion_matrix
from sklearn.metrics import confusion_matrix, roc_curve, roc_auc_score, accuracy_score, classification_report
import warnings
warnings.filterwarnings('ignore')
# Function to evaluate the model.
def evaluate_model(ytest, ypred, ypred_proba = None):
if ypred_proba is not None:
print('ROC-AUC score of the model: {}'.format(roc_auc_score(ytest, ypred_proba[:, 1])))
print('Accuracy of the model: {}\n'.format(accuracy_score(ytest, ypred)))
print('Classification report: \n{}\n'.format(classification_report(ytest, ypred)))
# In this notebook we don't load validation set, we load train and test because we have already decided the model, so now
# is the time to apply the train and test
x_train = pd.read_csv("../data/processed/X_train.csv", index_col=0)
y_train = pd.read_csv("../data/processed/y_train.csv", index_col=0)
x_test = pd.read_csv("../data/processed/X_test.csv", index_col=0)
y_test = pd.read_csv("../data/processed/y_test.csv", index_col=0)
# Import the preprocessor.
preprocessor = pickle.load(open('../models/preprocessor.pickle', 'rb'))
# Import the xgboost model.
xgb_model = pickle.load(open('../models/xgboost.pickle', 'rb'))
xgb_model = Pipeline([
('preprocessor', preprocessor),
('classifier', XGBClassifier(n_jobs = -1, random_state = 12345))
])
# Cross validation to choose the best parameters.
grid_param = {
'classifier__n_estimators': [300, 500],
'classifier__max_depth': [4, 8, 10],
'classifier__learning_rate': [0.05, 0.1],
'classifier__verbosity': [0, 2, 3]
}
CV = GridSearchCV(xgb_model, grid_param)
CV.fit(x_train, y_train)